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Abstract 

Background: Diagnostic problems in clinical trials are sometimes ordinal. For example, colon tumor staging was 
performed according to the TNM classification. However, clinical data are limited by markedly small sample sizes in 
some stage. 

Methods: We propose a distribution-free test for detecting ordered alternatives in a completely randomized design. 
The new statistic is based on summing all correctly (ascending) ordered samples. 

Results: The exact mean and variance of the null distribution are derived and it is shown that this distribution is 
asymptotically normal. Furthermore, we show using Monte Carlo simulation that the proposed test is a significant 
improvement over the Terpstra-Magel test. That is, power is decreased where the investigator falsely assumes an a 
priori ordering relationship. 

Conclusions: We conclude that these tests frequently detect an ordered trend when, in fact, one does not exist. 
However, the new test can reduce the error rate, at least not to the extent in which the Jonckheere-Terpstra test 
does. 



Background 

This paper focuses on considering nonparametric tests for 
the non-decreasing ordered alternative of k(>3) groups. 
The hypothesis to be tested is H 0 : Fi(x) = F 2 {x) = ■■■ = F k {x) 
for all x and H 1 : F^x) > F 2 (x) > ■ ■ ■ > F k (x), for all x with 
Fi{x) >F k {x) for some x, where F 1 {x),F 2 {x), ■■■,F k {x) are 
continuous distribution functions. 

In this article, we assume the location model with F^x) = 
F{x -ft - 9i), where ^ is a location parameter and 0,- repre- 
sents the effect of group i, i = 1, 2, k. This im- 
plies that the underlying populations may differ only 
in location. Throughout the article, let Xa, Xa, ...,Xi„ t , 
i = 1, 2, k represent independent random sam- 
ples from the k populations with distribution functions 
Ft (x), i = 1, 2, k, respectively. 

Nonparametric order restricted inference has been ex- 
tensively investigated in past literature and new studies 
are continuing to emerge. For instance, Puri [1], Puri 
and Sen [2], and Padmanabhan et al. [3] applied the con- 
cept of Chernoff-Savage-type statistics to nonparametric 
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ordered alternative tests. Studies that used power results 
to compare the validity of linear rank tests included 
Buning and Kossler [4], Beier and Biining [5], Biining 
and Kossler [6], Buning [7], Buning and Kossler [8], 
Buning and Kossler [9], Kossler [10] and Kossler [11]. 

The earliest and most classic treatment of k (> 3)-sam- 
ple distribution-free statistic for ordered alternatives was 
proposed by Jonckheere [12] and Terpstra [13]. The test is 
known as the Jonckheere-Terpstra test (hereafter referred 
to as the JT test) and is based on a sum of C 2 Mann- 
Whitney statistics (Mann and Whitney, [14]; Hollander 
and Wolfe, [15]). In order to define the JT statistic, we 
express the Mann- Whitney statistics as 

Ui m = l[xi jo x mj ^) , l<l<m<k, 

/<=1/„,=1 

where / (x, h , x mja ) = j £ if Xm >" , and the JT statistic 

k-l k 

is given by JT = ^ ^ Ulm ' 
1=1 m=l+l 

Other tests for ordered alternatives were developed by 
Cuzick [16] and Le [17]. Among the JT, Cuzick, and Le 
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tests, the results from Mahrer and Magel [18] did not 
establish any of the tests as having overwhelmingly 
higher power over the others across different location 
parameters. Neuhauser et al. [19] presented a modified 
version of the JT test (hereafter referred to as the MJT 
test). The form of the MJT statistic is identical to the JT 
test except that the Mann-Whitney statistic U[ m multi- 
plies the weight m - I as the new kernel. Study results 
showed that the MJT test often produced a higher power 
than the JT test for the ordered alternative. We also 
noted that Tryon and Hettmansperger [20] presented 
the JT and MJT tests as members of a more general class 
of nonparametric tests. In the test statistics described 
above the kernels of the tests are almost all derived by 
comparing two pairs of sample observations at a time. 
However, Terpstra and Magel [21] proposed a test (here- 
after referred to as the TM test) where the kernels of 
TM test are based on information obtained simultan- 
eously across all samples. The statistic is determined by 

k 

adding the Y[ n i indicator functions, that is, 

;=i 

«i j't 

/i=i ;»=i 

where l{x\ h <^ 2 / 2 ^ " ' - x kj k ) is equal to one, provided at 
least one strict inequality; otherwise, l(xy l <X2j 2 <---<X/ ( j k ) 
is equal to zero.. 

Terpstra et al. ([22] a, b) proposed a new nonparametric 
test statistic (hereafter referred to as the KTP test) that is 
a generalization of the TM test. The idea is to replace the 
indicator kernel from the TM test with Spearman's rank 
correlation coefficient, that is, 

«1 «i 

KTP = ■ ■ r (*Vi ' *% ' -' Xk h) > 

where is Spearman's rank correlation coefficient between 
the observed data and the corresponding group number. 
In this study, we propose a new test is based on the in- 

k 

formation present in the AT = J^J «, k-tuplets, where a 

;=i 

k-tuplet includes one observation from each treatment 
group. All correctly (ascending) ordered samples are then 
summed to form a statistic that is distributed approxi- 
mately as a normal distribution. Details of this new test 
and its asymptotic distribution are provided, and the com- 
putational algorithm is presented in the Additional file 1. 
A colon cancer data example is given in data example sec- 
tion. Finally, we present a finite sample simulation study 
which compares the proposed test, the JT test, MJT test, 
TM test, and the KTP test in terms of power. A computer 
program written in R that implements the proposed 



methods will be available from the first author upon re- 
quest. It is recommended that readers who are not inter- 
ested in the details of the computational algorithm skip 
the Additional file 1. 

Methods 

Test statistic 

The new nonparametric test for non-decreasing alterna- 
tives is based on the following statistic, 

HI «t 

Where k(xi,x 2 , ...,x k ) = '^.JjRjxi) = i), R W de- 
notes the rank of x t with respect to x 1( x 2 ,..., x k , and I(.) 
denotes the indicator function. 

The remainder of this section presents and derives re- 
sults pertaining to the null distribution of the proposed 
test statistic. We assume throughout this section that 
the observed data, {Xy} is essentially a random sample 
from some continuous probability distribution function 
F. Hence, the possibility of ties has a probability of zero. 
In principle the test statistic uses the k-tuplet method of 
Terpstra and Magel. Additionally, in the null hypothesis 
each k(xy l ,X2j 1 , ...,Xkj k ) follows the Binomial (k, 1/k) 
distribution. For these reasons, we will refer to this test 
as the KTMB test. 

The exact null distribution 

Let N denote the sum of the sample sizes for each treat- 
ment. Namely, let TV '= «i + ••■ + n k . Here, we have N\l 
(«i ! ■■■ n^l) partitions of the numbers 1, N. The null 
distribution of T means each one of these partitions is 
equally likely so the mean and variance can be calculated 
directly by multiplying each possible value of T with its 
probability. When the number of partitions is small, we 
can easily calculate the exact distribution by hand or 
with the computer. Table 1 shows the probabilities, 
means, and variances of the test statistic T for sample 
size arrangements (2, 1, 1), (2, 1, 2), and (1, 1, 3) 
respectively. 

Real world cases are not always as simple as the above 
illustration. For example, when k=4, «i = « 2 = «3 = «4 = 5, 
we have 20 !/(5 ! 5 ! 5 ! 5 !) = 1.1733 x 10 10 partitions. Such 
a distribution function cannot be calculated even with 
the most efficient personal computers. We will therefore 
introduce a Monte Carlo approximation to the null dis- 
tribution. On the other hand, if the distribution of T can 
be approximated or can be shown to converge to a well- 
known distribution, we can avoid computational com- 
plexity altogether. 
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Table 1 Some exact null distributions for the proposed test statistic 



Sample sizes Test statistic value (KTMB) Mean and variance 



(n,, n 2 , n 3 ) 


0 


1 


2 


3 


4 


5 


6 


7 


8 


9 


12 


e m 


vm 


(2, 1, 1) 


2/12 


3/12 


4/12 


1/12 


1/12 




1/12 










2 


2.667 


(2, 1,2) 


2/30 


2/30 


6/30 


4/30 


6/30 


2/30 


3/30 


2/30 


2/30 




1/30 


4 


6.867 


(1, 1,3) 


2/20 


3/20 


4/20 


6/20 


1/20 


1/20 


1/20 


1/20 




1/20 




3 


5.000 



The mean and variance 

If the asymptotic null distribution of a test statistic is 
normal and the exact mean and variance of T under H 0 
in standard form can be established we can then 
standardize T by using the exact mean and variance to 
obtain Z KTMB , where Z KTMB = {T-E 0 (T)}/y/V 0 (T). In 
this case, we can find critical values from the standard 
normal table. 

We will start by finding the mean value of T, E 0 (T), 

k 

noting that T is nothing but a sum of the k x J^J «, 

i=l 

Bernoulli (l//c) distribution. It is straightforward to get 



Eo(T) = n«, 

1=1 

Here, and in the following, we let n* 

k-1 

V 0 (T) = vl + J2v* + v 2 k 



(1) 



where vg = {l-(l/Ar)}JJ«j for no tie, v 



(2) 

n*k{k-l) 



{(/c-2)!//c!-l//c 2 } for k ties for i * j. 
For the case of i ties, we present an algorithm for the 



in Additional file 1. Readers 



k-i 

computation of the 2_. v l 

i=l 

who are not interested in the details of this algorithm 
may want to skip the Additional file 1 and go to data ex- 
ample section, in which examples based on real data are 
provided. 

The asymptotic null distribution 

In this section we will look to see if the asymptotic null 
distribution of test statistic T follows the standard nor- 
mal distribution. In other words, we prove that 



T-E 0 (T) D_ 

v^) 



AT(0,1) 



(3) 



H 0 will therefore be rejected for large values of T . The 
normal approximation for the procedure is to reject H 0 
if T* > Z\ _ „; otherwise do not reject H 0 . Note that the 



critical value Z\ _ a is chosen to make the Type I error 
probability equal to a. That is, a ~ P(T* > Z\ - a \Ho true). 
We note that (3) is a direct consequence of Theorem 1, 
which we now state. 

k 

Theorem 1 Let N = «/ and assume ^ = A; + o(l) 



l=i 



where X l e (0, 1) . Then, under Hq Tn = t Nk . 1/2 / J ' ' ' 

^N\Q,^kyX where X\ = X^X 2 ^ . 

Proof of Theorem 1 Terpstra and Magel [21] proved 
that TM statistic follows a normal distribution as sample 
sizes go to infinity by using projection technique from 
Hettmansperger and McKean ([23], p. 81). In what fol- 
lows all limits are taken with respect to N, as N— >°o. To 
apply their theorem to our case, let 

"l n k k 



k-N k ~ x l 2 



A=i 4=i *=i 



x(X im )+/ i (; i ^)---]= jv ^_ 



Zik(Xi m )- 



ZikiXi, 



F'- l (x)[l-F(x)] k - 1 , I = 1, 



where 



(M)!(*-Q! 



,i(l*t) 



7=1 

The projection of T N , say, P N can be defined as, 



k ni 
1=1 m=l 



(4) 



E 

i=i 



L n {l)^h~i 



N k-l/2 



i 



Zik{X, 



llll > 



E[Zi k {Xi m )} = A and V[Z lk {X lm )\ = a\ can be proved 
by Beta distribution. The convergence criteria on the k 
sample sizes imply that, 



L n (l)yfni 



A/*-!/2 



X* +o(l). 



(5) 
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It now follows from (4), (5), and limiting moment gen- 
erating function theory that, 



P N -^N[0,J2x}al 



(6) 



Let us now consider V [T N ], which we write as, 

-i «i "t «i n k 



VT 



il=l ( t =l; 1= l ; t =l 

xCOKf/c^ii,, ••■,^ i J,/c(x 1A , ■■■,x kh )]. 

Consider first the case of k ties for * j. it is straightfor- 
ward to show COV [k(x lh x kik ) , k (x Vl , • , x kjk )) = k 

(k-1) [^f^ ~f] • Next, consider the case in which 

there are exactly three ties among the different sub- 
scripts. For example, if we let R u denotes the rank of 
x u with respect to x lt x 2 ,..., X k , R v denotes the rank of 
x v with respect to x lt x 2 ,..., x 2k ^ 3 , u < v-k, R u < R v , and 
Xj, X 2 , X 3 denote the tied observations then the 
covariance term has the form COV [I u , IJ where, R u 
denotes the rank of x u with respect to X4,, ...jX^, Xi, 

X^+i, ...,X U , ...,Xi 2 , X2,X[ 2 + i, ...,X[ 3 , X^,Xi 3+ \...,X k 

and l u = I{R U = u), and 7?,, denotes the rank of x v with 
respect to X k+1 , ...,X/ f+ / 1 _ 3 ,X 1 ,X< r+ ; 1 _ 2 , ...,X k+ i 2 -3,X2, 

■■-,X V , ...,X*; + ; 3 _3,X3, X^ + / 3 _2, ...,X2k-3 and I v = 

I(R V = v). 

Under H 0 , E[I U ] = E[I V ] = j. Next, consider E [I U I V ]. 
This expectation contains 2/c-3 observations, so that 
under H 0 , each of the (2£-3)! permutations of the observa- 
tions are equally likely. However, there are only the num- 
bers of {1 : R u -l}n{l : R v -1} possible ways, say hs{Rw Rv)> 
plus the numbers of {R u +1 : 2k-3}n{l : R v -lj possible ways, 
say I LS (R U > R v )> plus the numbers of {R u +1 : 2k-3jn{R„+l : 
2k-3} possible ways, say Ill(R u > R v )> to preserve X h X 2 , and 
X 3 . Furthermore, there are C^, "' possible ways to 
preserve Xj,, ...^X^, X^+i, ...,X u -i to the left of X u , 

Ci L ! k { -i-{u-l) possible ways to preserve X k+1 , ..,X Wr3 , 
X k +k-2, ■■■,X k+ i 2 -3,X k+ i 2 - 2 , X v _i to the left of X w 
^-S£^V*-i-(»^)]} possible ways to preserve X u+1 , 
Xi 2 ,X h+u ...,X h , X h+1 ...,X k to the right of X w and 

^hdRuM-^k-u-hs+i+v-k-i-u+i) ~ 1 Possible way to pre- 
serve X v+1 , ...,X k+h _ 3 ,X k+h _ 2 , ..X 2k -3 to the right of X v . 
Hence, these arguments imply that, 



where ti + t2 + fe = 3, ti, t2, and t3 = 0, 1, 2, 3. 



Now, for a given l h l 2 , and l 3 , there are n^n^n^Y 

t=i 

[^(^-l)] 7 ^*' 1 ' 7 *'*' 2 ' 7 '**' 3 ' of these covariance terms. Next, 
consider all possible R u , R v and all possible treatment lo- 
cations {m and n), in the case of one tie X lt (7) reduces 
to, 



Ru=m R*=n 



\ K Z M* z /- i -ii-4 >-v_*_i_(„^i) ^k-u-{lLs(Ru,R,)-t2-lv-k-l-(u-4)]} _ 1 , 

(2/c-l)! F 

k 



where t 2 + t 2 + t 3 = 1, t lt t 2 , and t 3 = 0, 1. 

From (6) and (8) it follows that V[T N ] - V[P N ] 
Asymptotic normality results are attainable. 



(8) 



o(l). 



Patient characteristics 

The institutional review board of Chang Gung Memorial 
Hospital approved the present study. Detailed information 
about patients with colon cancer, such as patient- and 
tumor-related factors and follow-up status, was retrieved 
from the Colorectal Section Tumor Registry at Chang 
Gung Memorial Hospital, Taiwan. All the data in this 
registry were prospectively collected. 

Results and discussion 

Data examples 

Between January 2006 and December 2010, 154 con- 
secutive patients with histologically confirmed colonic 
adenocarcinoma underwent curative surgeries at the 
Chang Gung Memorial Hospital in Chiayi. The stage IV 
colon cancer, non-curative surgeries, rectal cancer and 
mucinous adenocarcinomawere excluded in this study. 
Tumor staging was performed according to the TNM 
classification described in the 6th edition of the cancer 
staging manual of the American Joint Committee on 
Cancer (Stage I, II, IIIA and IIIB). The different tumor 
staging require a different treatment to optimize patient 
and hospital outcomes. An ordinal logistic regression 
model was developed with predictors as follows: age, gen- 
der, tumor location, histologic differentiation, preoperative 



COV(I u ,I v 



3!(/c-4)!(^-4)!C^f;^ )+/ " (g " A)+/ " (g " A ^C^ A) - fl C; 
{2k-3)\ 



v-k-l-{u-4)^k-u-{I LS {R u ,R v )-l-[v-k-l-{u-4)\} 1 
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Table 2 Order restricted inference results for the colon 
cancer data 





JT 


MJT 


TM 


KTP 


KTMB 


Test Statistic 


3.01 


2.69 


1.44 


2.17 


1.29 


p-value 


0.00132 


0.00361 


0.07473 


0.01483 


0.09830 



albumin level, preoperative carcinoembryonic antigen 
level, and underlying medical illnesses. 

To illustrate the KTMB test, assume an outcome with 
four stages and a set of cases consisting of one case from 
each stage. The case from Stage I has risks of 0.50, 0.25, 
0.15 and 0.10 for Stage I, II, IIIA and IIIB, respectively. 
The case from Stage II has risks 0.26, 0.52, 0.17 and 
0.05; the case from Stage IIIA has risks 0.06, 0.32, 0.42 
and 0.20; the case from Stage IIIB has risks 0.12, 0.18, 
0.30 and 0.40. The risk for Stage IIIB (say, event) is 
higher for the case that belongs to this stage (0.40) than 
for the other cases (0.10, 0.05 and 0.20). The risk for 
event is second-highest for the case from Stage IIIA 
(0.20 versus 0.10, 0.05 and 0.40). However, the risk for 
event is lowest for the case from Stage II (0.05 versus 
0.10, 0.20 and 0.40). The risk for event is third-highest 
for the case from Stage I (0.10 versus 0.05, 0.20 and 
0.40). Therefore, the risks correctly identify the cases 



from Stage IIIA and IIIB but not Stage I and II, resulting 
in a score of 2 for this set (k(xi, X2, X3, #4)). 

Hence, the set of hypotheses was H 0 : Fj{x) = F n {x) = 
Fhia{x) = F IIIB {x) for all x and H y : Fj{x) > F u {x) > F njA 
(x) > F IIIB {x), where Fi{x) F nlB {x) for some x. 

Five test statistics and the corresponding p-values are 
given in Table 2. Since a plot of this data set in Figure 1 
exhibits a non-increased trend, it appears that the JT, 
MJT and KTP tests have falsely conclusion (p < 0.05). 
The KTMB test has the largest p-value (See Table 2). 
Moreover, Stage IIIB patients reported significantly more 
risk for Stage IIIB than Stage II subjects (ANOVA, post 
hoc: IIIB > II, p = 0.003) while Stage I, II and IIIA pa- 
tients did not differ in risk for Stage IIIB (ANOVA, post- 
hoc: p>0.05) Hence, we conclude that the risk for Stage 
IIIB do not increase with the patient's TNM in the 
model. That is, the discrimination performance of the 
ordinal logistic model is not very well between Stage I, II 
and IIIA. 

Comparison with respect to size and power 

To determine if the underlying population came from 
different skew and kurtosis distributions that impact on 
the power of the test statistic, we used log-F distribu- 
tions with combinations of 2, 4.5 and 10 degrees of 
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freedom to generate the random variable. We can there- 
fore define random variable Xy as: X,y = f5, + £y, where £ 2 y 
is the iid log-F distribution, and 0,- are location 
parameters. 

For the numbers of treatment (k), sample sizes («,■) 
and location parameters (0,-) we examine the different 
combinations of k = 3 and 4, ni = 4, 5, 8 and 10, 0, = 0, 
0.25, 0.5, 0.75, 1 and 1.25. We investigated designs under 
assumed alternatives which are of the forms of concave 
and convex. Programs to compare powers were written 
in R 2.9.2 (R Development Core Team, Vienna, Austria). 



The estimations were conducted by simulating 10,000 
different sets of samples. Furthermore, we estimated the 
power by counting the number of times H 0 was rejected 
and using the value to divide by 10,000. Ideally, we be- 
lieve that the test should have higher power than a gen- 
eral alternative test when Hi is true, and should have 
low power for any alternative that does not fit the profile 
given in Hi. 

In general, the JT and KTP tests have the highest pow- 
ers for the ordered alternative cases. Comparing with 
TM test, the gain percentage in power, DP = (KTMB - 



Table 3 Estimated powers and type I error rates of ordered tests under significance level 0.05 



Location parameter 


KTMB 


KTP 


JT 


MJT 


TM 


DP 


Location parameter 


KTMB 


KTP 


JT 


MJT 


TM 


DP 


n n =4, n 2 =4, n 3 =4 














^=10, n 2 =10, n 3 =5 
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TM)/TM, ranges from -6.29% to 11.27% with the aver- 
age gain percentage in power being 2.63% (difference of 
percentage). 

Consider the corresponding alternatives of the form of 
concave and convex shapes. The powers of the KTMB 
test outperforms (lower power) the KTP, JT, MJT, and 
TM tests when balanced design. The loss percentage in 
power, DP = (minimum of KTP, JT, MJT, and TM - 
KTMB) /KTMB, ranges from -7.28% to 20.00% with the 
average loss percentage in power being 4.25% for k = 3. 
The DP ranges from -13.2% to 27.68% with the average 
loss percentage in power being 8.83% for k = 4. 

When the sample sizes corresponding to the non- 
decreased trend location parameters are comparatively 
large, the KTMB test is better than KTP, MJT, JT, and 
TM tests. The DP ranges from -7.29% to 9.30% with the 
average loss percentage in power being 2.47% for k = 3. 
The DP ranges from -6.52% to 30.93% with the average 
loss percentage in power being 12.07% for k = 4. How- 
ever, the KTP test slightly better than KTMB test when 
the underlying population is skewed to the right (see 
Table 3). 

Based on the simulation results above, we conclude 
that the KTMB test is better than the TM test in regards 
to the power against ordered alternatives. Moreover, the 
KTMB test offers built in protection for the situation 
when an investigator falsely assumes an a priori ordered 
relationship. 

Table 3 just represent a small subset of the many dif- 
ferent scenarios that we simulated. For example, we also 
conducted simulations for numerous other alternative 
patterns. Interested persons may contact the corre- 
sponding author for these simulated results. 

Conclusions 

This research proposes a new nonparametric test for the 
ordered alternative problem. The new test statistic is 
based on the calculating all k(xy i ,X2j 2 , ...,x^) in proper 
(ascending) order. In other words, the new test statistic 
collects the information of each observation for each 
treatment to provide the message of "increasing" to the 
test statistics. A higher test statistics means a stronger "in- 
creasing" message. This is also why we expect the new test 
statistics to offer better power under certain situations. 

Due to the small number of groups and sample sizes, 
we tabulated and listed their distribution as well as the 
exact mean and variance of the null distribution. From 
the equation for the exact mean and variance of the null 
distribution was derived and the asymptotic null distri- 
bution is normal were given. 

We also use the example of ordinal risk prediction of 
colon cancer to compare the test statistics mentioned in 
the papers. A finite sample simulation study was also 



used to explore in-depth how the powers of JT MJT, 
TM, KTP and KTMB tests under different underlying 
populations, treatment numbers and sample sizes. Based 
on the example and simulation results, we conclude that 
these tests frequently detect an ordered trend when, in 
fact, one does not exist. However, the KTMB test can re- 
duce the error rate, at least not to the extent in which 
the JT and MJT tests do. 

Ben Van Calster et. al. extend the main measure of bin- 
ary discrimination, the c-statistic or area under the ROC 
curve, to nominal polytomous settings by polytomous dis- 
crimination index (PDI) [24] . They mention it is desirable 
that the risk of each group is highest for the case that be- 
longs to this group in a set of cases. Therefore, the PDI 
score awarded to a set equals the number of groups for 
which this holds. Based on this point of view, in our opin- 
ion, the KTMB test can not only be used for detecting the 
non-decreasing alternatives but can also be measured to 
summarize polytomous discrimination. 
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